Image Processing Apparatus And Image Sensing Apparatus

ABSTRACT

A tracking process portion includes a search area setting portion for setting a search area in the input image, an image analysis portion for analyzing an image in the search area, an auxiliary track value setting portion for setting an auxiliary track value based on a result of the analysis, a track value setting portion for setting an auxiliary track value based on a result of the analysis and deciding whether the set track value is correct or not, and a track target detection portion for detecting a track object from the image in the search area based on the track value. If the set track value is incorrect, the track value setting portion performs a switching operation for setting the auxiliary track value and a track value.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Japanese Patent Application No.2009-093976 filed on Apr. 8, 2009, which is hereby incorporated byreference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus forprocessing an input image and an image sensing apparatus including theimage processing apparatus.

2. Description of Related Art

Recent years, a digital image sensing apparatus for sensing andrecording images, and a digital reproduction apparatus for reproducingimages are widely available. Among these electronic apparatuses, thereis an apparatus that performs tracking process in which a predeterminedsubject (hereinafter referred to as a track object) is detected frominput images supplied sequentially. A result of the detection can beused for processing images that are taken and recorded or images thatare reproduced, or for controlling various parameters such as a focalpoint, exposure and the like in image sensing.

However, it is difficult to detect the track object continuously amonginput images changing from moment to moment while maintaining highaccuracy.

Therefore, there is proposed a method of recognizing a plurality ofcolors of the track object together with a positional relationship orthe like of each color part, so as to detect the track object based on aresult of the integral recognition thereof. If the track object isdetected by this method, the track object can be detected accurately.

However, the above-mentioned detection method of the track objectrequires to process much information at one time. Therefore, there is aproblem that the process becomes complicated so that the process timeand power consumption are increased. In addition, there is also aproblem that the detection becomes difficult if the part for detectingthe track object is blocked by an obstacle.

SUMMARY OF THE INVENTION

An image processing apparatus of the present invention includes:

a track value setting portion which sets a track value which is a signalvalue indicating a track object in an input image;

an auxiliary track value setting portion which sets an auxiliary trackvalue which is a signal value indicating the track object and isdifferent from the track value; and

a track target detection portion which detects a pixel having the trackvalue from the input image, wherein

the track value setting portion is capable of performing a switchingoperation in which the auxiliary track value is set as a new track valueinstead of the set track value.

An image sensing apparatus according to the present invention includes:

an image sensing portion which generates an input image by imagesensing;

the above-mentioned image processing apparatus, wherein

the image sensing apparatus performs control based on a result of thedetection of the track object by the image processing apparatus.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a structure of an image sensingapparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a structure of a tracking processportion of the image sensing apparatus according to an embodiment of thepresent invention.

FIG. 3 is a flowchart illustrating an operation of the tracking processportion of the image sensing apparatus according to an embodiment of thepresent invention

FIG. 4A is a diagram illustrating an example of an input image.

FIG. 4B is a diagram illustrating a result of image analysis of theinput image illustrated in FIG. 4A.

FIG. 5 is a flowchart illustrating an example of a setting method of anauxiliary track value.

FIG. 6A is a diagram illustrating an example of the input image in thecase where the track value is correct.

FIG. 6B is a diagram illustrating a result of image analysis of theinput image illustrated in FIG. 6A.

FIG. 7A is a diagram illustrating an example of the input image in thecase where the track value is incorrect.

FIG. 7B is a diagram illustrating a result of image analysis of theinput image illustrated in FIG. 7A.

FIG. 8A is a diagram illustrating an example of the input image of thenext frame of the input image illustrated in FIG. 7A.

FIG. 8B is a diagram illustrating a result of image analysis obtainedfrom the input image illustrated in FIG. 8A.

FIG. 9A is a diagram illustrating an example of the input image forillustrating a first variation example.

FIG. 9B is a diagram illustrating a result of image analysis of theinput image illustrated in FIG. 9A.

FIG. 10A is a diagram illustrating an example of the input image forillustrating a second variation example

FIG. 10B is a diagram illustrating a result of image analysis of theinput image illustrated in FIG. 10A.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Meanings and effects of the present invention will be further clarifiedfrom the following description of embodiment. However, the followingembodiment is merely one of embodiments of the present invention, andmeanings of the present invention and individual elements are notlimited to those described in the following embodiment.

An embodiment of the present invention will be described with referenceto the attached drawings. First, an example of an image sensingapparatus according to the present invention will be described. Notethat the image sensing apparatus described below is capable of recordingsound, moving images, and still images, like a digital camera.

<<Image Sensing Apparatus>>

First, a structure of the image sensing apparatus will be described withreference to FIG. 1. FIG. 1 is a block diagram illustrating a structureof the image sensing apparatus according to an embodiment of the presentinvention.

As illustrated in FIG. 1, the image sensing apparatus 1 includes animage sensor 2 constituted of a solid-state image sensor such as acharge coupled device (CCD) or a complementary metal oxide semiconductor(CMOS) sensor for converting an input optical image into an electricsignal, and a lens portion 3 for forming an optical image of a subjectin the image sensor 2 and adjusting light quantity and the like. Thelens portion 3 and the image sensor 2 constitute the image sensingportion, and an image signal is generated by the image sensing portion.Note that the lens portion 3 includes various lenses (not shown) such asa zoom lens and a focus lens, and an iris stop (not shown) for adjustingquantity of light entering the image sensor 2.

Further, the image sensing apparatus 1 includes an analog front end(AFE) 4 for converting the image signal that is an analog signal outputfrom the image sensor 2 into a digital signal and adjusting a gain, asound collecting portion 5 for converting input sound into an electricsignal, an image processing portion 6 for converting image signalconstituted of red (R), green (G) and blue (B) digital signals outputfrom the AFE 4 into a signal constituted of a luminance signal (Y) andcolor difference signals (U and V) and for performing various imageprocessings on the image signal, an audio processing portion 7 forconverting an audio signal that is an analog signal output from thesound collecting portion 5 into a digital signal, a compressionprocessing portion 8 for performing a compression coding process forstill images such as the JPEG (Joint Photographic Experts Group)compression method on the image signal output from the image processingportion 6 and performing a compression coding process for moving imagessuch as the MPEG (Moving Picture Experts Group) compression method onthe image signal output from the image processing portion 6 and on theaudio signal from the audio processing portion 7, an external memory 10for recording compression coded signal after the compression codingprocess by the compression processing portion 8, a driver portion 9 forrecording and reading the image signal in and from the external memory10, and an expansion processing portion 11 for expanding and decodingthe compression coded signal read from the external memory 10 by thedriver portion 9. In addition, the image processing portion 6 includes atracking process portion 60 for performing a tracking process ofdetecting a track object from an input image signal. Note that a detailof a structure of the tracking process portion 60 will be describedlater.

In addition, the image sensing apparatus 1 includes an image outputcircuit portion 12 for converting the image signal decoded by theexpansion processing portion 11 into a signal in a faun that can bedisplayed on a display apparatus (not shown) such as a display, and anaudio output circuit portion 13 for converting the audio signal decodedby the expansion processing portion 11 into a signal in a form that canbe reproduced by a reproduction apparatus (not shown) such as a speaker.

In addition, the image sensing apparatus 1 includes a central processingunit (CPU) 14 for controlling operations of the entire image sensingapparatus 1, a memory 15 for storing programs for performing theprocesses and temporarily storing signals when the programs areexecuted, an operating portion 16 for entering instructions from a user,such as a button for starting to take images or a button for decidingvarious setting, a timing generator (TG) portion 17 for delivering atiming control signal for synchronizing operation timings of theindividual portions, a bus line 18 for communicating signals between theCPU 14 and the individual portions, and a bus line 19 for communicatingsignals between the memory 15 and the individual portions.

Note that any type of external memory 10 can be used as long as it canrecord the image signal and the audio signal. For instance, asemiconductor memory such as a secure digital (SD) card, an optical discsuch as a DVD, and a magnetic disk such as a hard disk can be used asthe external memory 10. In addition, the external memory 10 may bedetachable from the image sensing apparatus 1.

Next, a fundamental action of the image sensing apparatus 1 will bedescribed with reference to FIG. 1. First, the image sensing apparatus 1performs photoelectric conversion of incident light from the lensportion 3 in the image sensor 2 so as to obtain the image signal that isan electric signal. Then, the image sensor 2 outputs the image signal tothe AFE 4 sequentially at a predetermined frame period (e.g., 1/30seconds) in synchronization with the timing control signal supplied fromthe TG portion 17. Then, the image signal that is a digital signalconverted from an analog signal by the AFE 4 is supplied to the imageprocessing portion 6. The image processing portion 6 converts the imagesignal into a signal using YUV and performs various image processingssuch as gradation correction and edge enhancement. In addition, thememory 15 works as a frame memory so as to hold the image signaltemporarily when the image processing portion 6 performs processes.

In addition, on this occasion, based on the image signal supplied to theimage processing portion 6, the lens portion 3 adjusts positions ofvarious lenses so that focus adjustment is performed, and adjusts anopening degree of the iris stop so that exposure adjustment isperformed. The various adjustments such as focus adjustment and exposureadjustment are performed automatically based on a predetermined programto be optical states or are performed manually based on an instructionfrom a user. In addition, the tracking process portion 60 performs thetracking process on the image signal supplied to the image processingportion 6. Note that a detail of operations of the tracking processportion 60 will be described later.

When recording a moving image, not only an image signal but also anaudio signal is recorded. The audio signal converted into an electricsignal and is output by the sound collecting portion 5 is supplied tothe audio processing portion 7 and is digitized, and processes such asnoise reduction are performed on it. Then, the image signal output fromthe image processing portion 6 and the audio signal output from theaudio processing portion 7 are both supplied to the compressionprocessing portion 8, and the image signal is compressed by apredetermined compression method in the compression processing portion8. In this case, the image signal and the audio signal are temporallyassociated with each other, so that the image and the sound are notdeviated from each other when they are reproduced. Then, the compressedimage signal and audio signal are recorded in the external memory 10 viathe driver portion 9.

On the other hand, when recording a still image or only sound, the imagesignal or the audio signal is compressed by a predetermined compressionmethod in the compression processing portion 8 and is recorded in theexternal memory 10. Note that it is possible that the image processingportion 6 performs different processes for recording a moving image andfor recording a still image.

The compressed image signal and audio signal recorded in the externalmemory 10 is read out to the expansion processing portion 11 based on aninstruction from a user. The expansion processing portion 11 expands thecompressed image signal and audio signal so as to deliver the imagesignal and the audio signal to the image output circuit portion 12 andthe audio output circuit portion 13, respectively. Then, the imageoutput circuit portion 12 and the audio output circuit portion 13convert the signals into signals of forms that can be displayed andreproduced by the display device and the speaker so as to output thesignals.

Note that the display device and the speaker may be integral with theimage sensing apparatus 1 or may be separate from the same so that theyare connected by using terminals provided to the image sensing apparatus1 and a cable or the like.

In addition, in a so-called preview mode in which a user can check theimage displayed on the display device without recording the imagesignal, it is possible to output the image signal from the imageprocessing portion 6 without compressing to the image output circuitportion 12. In addition, it is possible that the tracking processportion 60 performs the tracking process in the preview mode regardlessof whether the image to be recorded is a moving image or a still image.

In addition, when recording a image signal of a moving image, inparallel to compressing by the compression processing portion 8 andrecording in the external memory 10, it is possible to output the imagesignal to the display device or the like via the image output circuitportion 12.

<<Tracking Process Portion>>

Next, a structure of the tracking process portion 60 illustrated in FIG.1 will be described with reference to the drawings. FIG. 2 is a blockdiagram illustrating a structure of the tracking process portion of theimage sensing apparatus according to an embodiment of the presentinvention. Note that for a concrete description, the image signal thatis supplied to the tracking process portion 60 for performing thetracking process is expressed as the image and is referred to as the“input image” in the following description. In addition, the subject tobe tracked is referred to as a track object, and a part of the trackobject, which is a characteristic part to be detected by the trackingprocess portion 60, is referred to as a track target.

The tracking process portion 60 includes a search area setting portion61 for setting a search area in the input image so as to generate andoutput search area information, an image analysis portion 62 foranalyzing images in the search area indicated by the search areainformation of the input image so as to generate and output imageanalysis information, an auxiliary track value setting portion 63 forsetting an auxiliary track value based on the image analysis informationif necessary so as to generate and output auxiliary track valueinformation, a track value setting portion 64 for setting a track valuebased on the auxiliary track value information and the image analysisinformation so as to generate and output track value information, and atrack target detection portion 65 for detecting a track target bydetecting a part to be the track value indicated by the track valueinformation from images in the search area indicated by the search areainformation of the input image so as to generate and output track targetinformation.

Track target specifying information supplied externally and track targetinformation delivered from the track target detection portion 65 aresupplied to the search area setting portion 61 and are used for settingthe search area. In addition, the track value setting portion 64 storesthe track value that is set based on the image analysis information andthe auxiliary track value indicated by the auxiliary track valueinformation in itself or in the memory 15 or the like so as to switchthe track value in accordance with the image analysis information.

In addition, the track target information output from the track targetdetection portion 65 indicates a position of the track target in theinput image (i.e., position of the track object). The image sensingapparatus 1 performs various processes based on the track targetinformation. For instance, setting control of the image sensing portionsuch as focus and exposure, image processing of the input image, and thelike are performed.

In addition, an operation of the tracking process portion 60 will bedescribed with reference to the drawings. FIG. 3 is a flowchartillustrating an operation of the tracking process portion of the imagesensing apparatus according to an embodiment of the present invention.

As illustrated in FIG. 3, the tracking process portion 60 first obtainsthe input image (STEP 1) and specifies the track target (STEP 2). Notethat obtaining of the input image in STEP 1 may be repeated until thetrack target is specified in STEP 2 so that the track target isspecified from the latest input image.

Specifying of the track target in STEP 2 may be performed, for example,by a user who checks the input image displayed on the display device ofthe image sensing apparatus 1 and specifies one directly, or by aprogram or the like automatically. In addition, it may be performed by auser who selects one from a plurality of track target candidatesspecified by a program or the like.

When a user specifies the track target, the user may operates theoperating portion 16 constituted of a cursor key, a touch panel, or thelike to as to specify the track target, for example. In addition, if thetrack target or the candidate thereof is specified by a program, it ispossible, for example, to perform a face detection process for detectinga nonspecific face in the input image or a face recognition process fordetecting a specific face, so as to specify a part of the track objecthaving the detected face (e.g., a body region that is a region existingin the direction from the middle of the forehead toward the mouth of thedetected face) as the track target or the candidate thereof. As a methodfor the face detection or the face recognition, it is possible to usevarious well-known techniques. For instance, it is possible to utilizeAdaboost (Yoav Freund, Robert E. Schapire, “A decision-theoreticgeneralization of on-line learning and an application to boosting”,European Conference on Computational Learning Theory, Sep. 20, 1995) forcomparing a weight table generated from a large volume of teachingsamples (face and non-face sample images) with the input image so as toperform the face detection or the face recognition.

The information of the track target specified by the above-mentionedmethod is supplied as the track target specifying information to thesearch area setting portion 61. The search area setting portion 61 setsthe search area round the track target specified by the track targetspecifying information (STEP 3). For instance, a region of apredetermined area round the track target is set as the search area.Note that the track target specifying information may include a positionof the track target (e.g., a barycenter position) and a size of thetrack target. Further, the search area may be a region having a sizecorresponding to a size of the track target (e.g., a size in which thetrack target can be sufficiently included) round the barycenter positionof the track target. In addition, the search area may have any shape.For instance, it has a rectangular shape as described later, or acircular shape or an elliptic shape.

The search area set in STEP 3 is output as the search area informationfrom the search area setting portion 61. Then, the image analysisportion 62 analyzes the image in the search area indicated by the searcharea information (STEP 4). For instance, a histogram of signal values ofpixels included in the search area (values of the same type as the trackvalue) is generated for the analysis. An example of a result of theimage analysis using the histogram (frequency distribution) will bedescribed with reference to FIGS. 4A and 4B. FIG. 4A is a diagramillustrating an example of the input image, and FIG. 4B is a diagramillustrating a result of image analysis of the input image illustratedin FIG. 4A. In particular, FIG. 4A is a diagram illustrating an exampleof the input image in which the search area is set, and FIG. 4B is adiagram illustrating a histogram of signal values obtained from theimage in the search area illustrated in FIG. 4A.

The track value and the signal value may be any type of value (e.g.,individual values of RGB, a luminance value, or a value of H of thesignal expressed by H (hue), S (chroma saturation) and V (brightness)).Hereinafter, for a specific description, an example of the case where ahue value (hereinafter referred to as color simply) is used will bedescribed. In addition, in the following description, the hue value isexpressed by using not a value of angle or the like but a simple colortype (e.g., yellow, green, blue and the like) in a simplified manner.

In an input image 40 illustrated in FIG. 4A, a part of pants and itsperiphery of a person A1 as a track object is specified as a tracktarget, so that a rectangular search area 41 is set in the part. Inaddition, the color of the pants of the person A1 is yellow Ye, and thecolor of a shirt is green G. Further, the input image 40 also includes aperson A2, and colors of pants and a shirt of the person A2 are bothblue B. In this case, the histogram of colors of individual pixels inthe search area 41 becomes as illustrated in FIG. 4B. Specifically, theobtained histogram indicates that the number of pixels of yellow Ye islargest and the number of pixels of green G is the next. The imageanalysis portion 62 generated such histogram information and outputs itas image analysis information.

The track value setting portion 64 sets the track value based on thehistogram indicated by the image analysis information (STEP 5). Thetrack value setting portion 64 sets, for example, the color having ahighest frequency in the histogram (yellow Ye in FIG. 4B) as the trackvalue. Note that it is possible to set the color specified by a user asthe track value. In addition, the set track value is held in the trackvalue setting portion 64 or in the memory 15.

When the track value is set in STEP 5, the input image of the next frameis obtained (STEP 6). Then, the search area setting portion 61 sets thesearch area in the obtained input image similarly to STEP 3 (STEP 7).For instance, the search area is set based on a position or a size ofthe track target specified or detected in the just previous input image.Then, the image analysis portion 62 analyzes the image in the searcharea of the input image similarly to STEP 4, so as to output the imageanalysis information (STEP 8).

Here, if the auxiliary track value is not set (NO in STEP 9), theauxiliary track value setting portion 63 sets the auxiliary track valuebased on the image analysis information (STEP 10). Then, the track valuesetting portion 64 outputs the currently set track value (e.g., yellowYe) as the track value information to the track target detection portion65. The track target detection portion 65 detects the pixel having thetrack value in the search area of the input image so as to detect thetrack target (STEP 13).

An example of a setting method of the auxiliary track value in STEP 10will be described with reference to the drawings. FIG. 5 is a flowchartillustrating an example of the setting method of the auxiliary trackvalue. As illustrated in FIG. 5, the auxiliary track value settingportion 63 first checks whether or not a candidate value exists based onthe histogram as illustrated in FIG. 4B, for example (STEP 101). Thecandidate value is a signal value having a highest frequency in thehistogram or a signal value having a frequency higher than apredetermined threshold value except for the currently set track value(yellow Ye). In the example illustrated in FIG. 4B, the green Gcorresponds to the candidate value. If there is no candidate value (NOin STEP 101), the process flow ends without setting the auxiliary trackvalue. On the other hand, if there is a candidate value (YES in STEP101), it is checked whether or not the candidate value is confirmed ntimes continuously (STEP 102). If it is not confirmed n timescontinuously (NO in STEP 102), the process flow ends without setting theauxiliary track value. If it is confirmed n times continuously (YES inSTEP 102), the candidate value is set as the auxiliary track value (STEP103). Then, the auxiliary track value information is delivered to thetracking setting portion 64, and the process flow ends. Note that FIG. 5indicates a process with respect to one input image, and the operationin FIG. 5 is repeated until the auxiliary track value is set in STEP103. In addition, n is a natural number.

On the other hand, if the auxiliary track value is set (YES in STEP 9),the track value setting portion 64 decides whether the currently settrack value is correct or not based on the image analysis information(STEP 11). An example of the method of deciding whether the track valueis correct or not will be described with reference to the drawings. FIG.6A is a diagram illustrating an example of the input image in the casewhere the track value is correct, and FIG. 6B is a diagram illustratinga result of image analysis of the input image illustrated in FIG. 6A.FIG. 7A is a diagram illustrating an example of the input image in thecase where the track value is incorrect, and FIG. 7B is a diagramillustrating a result of image analysis of the input image illustratedin FIG. 7A. Note that FIGS. 6A, 6B, 7A and 7B are the same as those ofFIGS. 4A and 4B illustrating an example of the input image and a resultof image analysis of the input image. In other words, the persons A1 andA2 are included in the input images 60 and 70, the person A1 is thetrack object, and the set track value is the yellow Ye. Note that it issupposed that the green G is set as the auxiliary track value in FIGS.6A, 6B, 7A and 7B.

In the input image 60 illustrated in FIG. 6A, the person A2 enters inthe search area 61 so as to block a part of the pants of the yellow Yethat is the track value. However, in the histogram illustrated in FIG.6B, the frequency of the yellow Ye that is the track value is stillhighest, and the frequency of the yellow Ye is higher than the frequencyof the green G of the auxiliary track value. In this case, it is decidedthat the currently set track value is correct (YES in STEP 11), so thatdetection of the track target is performed without switching the trackvalue (STEP 13).

In contrast, in the input image 70 illustrated in FIG. 7A, the person A2enters in a search area 71 so as to block a major part of the pants ofthe yellow Ye that is the track value. Therefore, in the histogramillustrated in FIG. 7B, the frequency of the yellow Ye that is the trackvalue is no longer highest. On the other hand, the frequency of thegreen G that is the auxiliary track value maintains a certain size to bedetected easily. In this case, the currently set track value is decidedto be incorrect (NO in STEP 11), and the auxiliary track value isswitched to the track value (STEP 12). In other words, the track valueis switched to the green G and is set. Then, the newly set track valueis used so as to detect the track target (STEP 13).

In STEP 13, the track target detection portion 65 detects the tracktarget by deciding whether or not a signal value of each pixel is thetrack value, so that track target information is output. For instance,various known algorithm such as ISODATA (Interactive Self Organizationof Data) method may be used for classifying into a group in which thesignal value becomes the track value and a group in which it does notbecome the track value, so as to detect the track target. In this case,for example, a plurality of center values may be given so that thesignal values are temporarily classified based on which center value isclose. Then, incorrect group (in which belonging signal values arelittle or the variance is large) may be eliminated (combined or split)while a new center value is set from signal values of groups after thetemporary classification, and further the temporary classification isrepeated for performing classification. Note that it is also possible toclassify signal values of pixels by setting the track value andsimilarity indicating the range that can be regarded as the track value.In addition, it is possible to set the barycenter position of pixels inthe group in which the signal value is regarded as the track value asthe position of the track target. It is also possible to set the regionin which pixels of the group in which the signal value is regarded asthe track value extend as a size of the track target. In addition, it ispossible to include these pieces of information in the track targetinformation.

After the track target is detected in STEP 13, it is checked whether ornot the tracking process is finished (STEP 14). If an instruction tofinish the tracking process is input from a user or the like (YES inSTEP 14), the tracking process is finished. On the other hand, if theinstruction to finish the tracking process is not input (NO in STEP 14),the process flow goes back to STEP 6 in which the input image of thenext frame is obtained, and the above-mentioned process (STEP 7 to STEP13) is performed on the input image. In this way, the tracking processis performed on the input images that are obtained sequentially.

Here, an example of the tracking process on the input image of the nextframe will be described with reference to FIGS. 8A and 8B. FIG. 8A is adiagram illustrating an example of the input image of the next frame ofthe input image illustrated in FIG. 7A. FIG. 8B is a diagramillustrating a result of image analysis of the image obtained from theinput image illustrated in FIG. 7A. Note that the above-mentioned casewhere the detection of the track target is performed by supposing thatthe track value is the green G with respect to the image in the searcharea 71 of the input image 70 illustrated in FIG. 7A will be described.

In the case illustrated in FIGS. 7A and 7B, if the detection of thetrack target is performed by supposing that the track value is the greenpixels of upper portion in the search area 71 are mainly detected aspixels indicating the track target. If the search area is set round thedetected track target in the same manner as the above-mentioned method,a search area 81 that is set with respect to an input image 80illustrated in FIG. 8A is positioned at the upper position than thesearch area 71 illustrated in FIG. 7A. In other words, the set searcharea 81 becomes close to the portion of the shirt in which pixelsindicating the green G that is the track value are concentrated.

When the search area 81 as illustrated in FIG. 8A is set, a frequency ofthe green G that is the newly set track value increases as the histogramillustrated in FIG. 8B. Therefore, it is possible to detect the tracktarget accurately. Note that the auxiliary track value in the followingtracking process is also set in the same manner as STEP 10. Then, if theset track value (green G) becomes incorrect (becomes difficult to bedetected), the switching operation is performed so that the setauxiliary track value is set as the track value.

As described above, the tracking process portion 60 of this example setsthe auxiliary track value adding to the track value that is set fordetecting the track target, so that the auxiliary track value can beswitched to be the track value. Thus, if the track target is blocked bya certain object and it becomes difficult to detect pixels of the settrack value, the auxiliary track value is switched to be the track valueso as to detect another track target (i.e., another characteristic partof the same track object). Therefore, it is possible to perform accuratedetection successively.

In addition, the track value and the auxiliary track value are not usedsimultaneously for the tracking process, but they are switched ifnecessary. Therefore, it is possible to suppress an increase ofcomputing amount necessary for detecting the track target. Therefore, itis possible to realize a high speed operation and low power consumption.

Note that the parameter n when setting the auxiliary track valueillustrated in FIG. 5 may be any numeric value, but it is preferable toset appropriately from the following viewpoint. If the value of n islarge, the signal value included in the search area successively andcontinuously for long period becomes the auxiliary track value.Therefore, it is possible to increase probability that the auxiliarytrack value indicates the characteristic part of the track object. Inaddition, it is possible to set a signal value that is resistant tovariation of an imaging situation so as to detect accurately as theauxiliary track value. On the other hand, if the value of n is small, itis possible to set the auxiliary track value rapidly. Therefore, it ispossible to suppress occurrence of a situation that the auxiliary trackvalue has not been set when switching of the track value becomesnecessary. Note that it is possible to adopt a structure in which n canbe changed in accordance with the situation.

In addition, concerning the decision whether or not the track value iscorrect in STEP 11 of FIG. 3, it is possible to decide that the trackvalue is incorrect so as to switch the track value of STEP 11, if thefrequency of the track value becomes not maximum or becomes smaller thanthe predetermined value, and the frequency of the auxiliary track valuebecomes maximum or becomes larger than the predetermined value. Inaddition, it is possible to decide that the track value is incorrect ifthe frequency of the auxiliary track value is larger than the frequencyof the track value. In addition, it is possible to decide correct orincorrect by considering whether or not the track value or the auxiliarytrack value is outstandingly larger than the frequency of the signalvalues in the periphery (i.e., whether it is easily detected or not).The decision of the incorrect may be performed in any manner. But ingeneral, it is decided that the track value is incorrect when the trackvalue becomes not a dominant value in the search area and the auxiliarytrack value becomes a dominant value.

In addition, it is possible to set a plurality of auxiliary trackvalues. For instance, if there are a plurality of candidate values, itis possible to set all or some of the plurality of candidate values asthe auxiliary track value. In addition, it is possible to assign theorder of priority to the plurality of set auxiliary track values. It isalso possible to select the auxiliary track value that is mostappropriate for the track value to be the track value on the stage inwhich the track value is switched.

First Variation Example

In the example described above, the case where only the track value andthe auxiliary track value are set is described, but it is possible toadopt a structure in which another value is further set. An example ofthis case will be described with reference to FIGS. 9A and 9B. FIG. 9Ais a diagram illustrating an example of the input image for describing afirst variation example, and FIG. 9B is a diagram illustrating a resultof image analysis of the input image illustrated in FIG. 9A. FIGS. 9Aand 9B are similar to FIGS. 4A and 4B illustrating an example of theinput image and a result of image analysis thereof. In other words, aninput image 90 includes the persons A1 and A2, the person A1 is thetrack object, the track value is the yellow Ye, and the auxiliary trackvalue is the green G. In addition, it is supposed that the background inthis example is a uniform hue value that is a cyan Cy.

In the case of this example, as illustrated in FIG. 9B, for example, theimage analysis portion 62 generates not only the histogram of a searcharea 91 but also the histogram of a background region 92. In addition,the auxiliary track value setting portion 63 sets the auxiliary trackvalue based on the histograms. Note that, as illustrated in FIG. 9A, thebackground region 92 may be a region which includes the search area 91and its peripheral region and has substantially the same center as thebackground region 92.

In the histogram of the search area 91 illustrated in FIG. 9B, a signalvalue having a largest frequency is the yellow Ye, a signal value havinga next largest frequency is the cyan Cy, and a signal value having afurther next largest frequency is the green G. Here, the cyan Cy thathas a second largest signal value is the candidate value. If it isconfirmed n times successively, the cyan Cy that is a signal value ofthe background is set as the auxiliary tracking color (STEP 101 to STEP103 in FIG. 5). Then, if the cyan Cy that is a signal value of thebackground is switched to the track value and is set (STEP 12 in FIG.3), the track target detection portion 65 detects the background, and itbecomes difficult to detect the track object A1.

Therefore, in this example, when the auxiliary track value is set, thehistogram of the background region 92 is referred to, so that the signalvalue of the background (hereinafter referred to as a background value)is determined while the background value is excluded from the candidatevalue. In the case illustrated in FIG. 9B, the cyan Cy that is a signalvalue having a largest frequency in the histogram illustrated in thebackground region 92 is set as the background value and is excluded fromthe candidate value.

With the structure of this example, it is possible to prevent thebackground value from being set as the auxiliary track value. Therefore,incorrect detection of the background by the track target detectionportion 65 is suppressed, so that accuracy of the tracking process canbe improved.

Note that it is possible to adopt the method of this example whensetting the track value in STEP 5 of FIG. 3. In particular, when thesignal value having a largest frequency in the search area 91 is set asthe track value, incorrect setting of the background value as the trackvalue can be suppressed. Therefore, if the method of this example isadopted in this case for setting the track value, it is appropriate.

In addition, although the background region 92 has the rectangular shapesimilarly to the search area 91, it is possible that the backgroundregion 92 has a shape different from that of the search area 91. Inaddition, the setting method of the background region 92 with respect tothe input image 90 is not limited to the case of the example illustratedin FIG. 9A, and it may be set in any way. For instance, the backgroundregion may be set by another method, like a whole region of the inputimage 90, a whole region except the search area 91 of the input image90, a whole region except a predetermined middle region of the inputimage 90, or the like.

Second Variation Example

In the above-mentioned example, the case where both the track value andthe auxiliary track value are hue value is described, but it is possiblethat they have different signal values. An example of this case will bedescribed with reference to FIGS. 10A and 10B. FIG. 10A is a diagramillustrating an example of the input image for describing a secondvariation example, and FIG. 10B is a diagram illustrating a result ofimage analysis of the input image illustrated in FIG. 10A. FIGS. 10A and10B are similar to FIGS. 4A and 4B illustrating an example of the inputimage and a result of image analysis thereof. In other words, an inputimage 100 includes the persons A1 and A2, the person A1 is the trackobject, and the track value is yellow Ye.

In the case of this example, the image analysis portion 62 generates aplurality of histograms having different types of signal values. Forinstance, as illustrated in FIG. 10B, a histogram of the hue value and ahistogram of the luminance value are generated. Further, in this case,if the type of the signal value of the track value is the hue value, thesignal value type of the auxiliary track value is the luminance value.

In this example, the track value and the auxiliary track value can beset independently based on each histogram. For instance, it is possibleto set the signal value having a largest frequency in each histogram asthe track value or the auxiliary track value, or to set the signal valuehaving a frequency that is outstanding from peripheral signal values asthe track value or the auxiliary track value. In addition, it ispossible to use different setting methods for setting the track valueand the auxiliary track value in accordance with a property of thehistogram to be generated (i.e., a type of the signal value). Inaddition, when the auxiliary track value is set, it is possible to setthe signal value that has been confirmed n times successively asdescribed above as the auxiliary track value.

With the structure of this example, if the detection of the track targetusing a certain type of signal value is difficult, it is possible toswitch to the detection using another type of signal value. Forinstance, if it is difficult to specify the luminance value of the tracktarget under the exposure environment where the luminance value changesactively, it is possible to switch to the detection using the hue value.Further, for example, if the hue value of the track object issubstantially the same as the hue value of the periphery so that thedetection of the track target is difficult, it is possible to switch tothe detection using the luminance value. Therefore, it is possible toperform the detection accurately in various exposure environments.

Other Variation Examples

Note that the signal value type that can be set as the track value andthe auxiliary track value is not limited to the hue value. For instance,the luminance value, any one of RGB values, a combination of RGB values,or any other type of signal value may be used.

In addition, it is possible to make the pixel signal value, the trackvalue or the auxiliary track value to be low gradation. If they are madeto be low gradation, a slight difference of signal value can be ignored,so that the track target can be detected easily. In addition, it ispossible to determine the degree of low gradation in accordance with thesignal value type to be used for the detection.

In addition, it is possible to crop the image based on track targetinformation output from the tracking process portion 60. For instance,it is possible that the image sensing portion generates a wide-angleinput image and that the image processing portion 6 crops apredetermined region including the track target (track object) from theinput image so as to generate a desired composition of image. If theimage cropping is performed in this way, it is sufficient for the userto direct the image sensing apparatus 1 toward the track object whoseimage should be taken in a simplified manner, so that a desiredcomposition of image can be obtained. Therefore, it is possible toreduce necessity of the user to concentrate on taking the image.

In addition, although the case where the image sensing apparatusperforms the tracking process (when the image is taken) is described, itis possible that the reproduction apparatus performs the same (when theimage is reproduced). For instance, this example can be applied to thecase where the track target is detected from the image to be reproduced,and the reproduction is performed by performing the image processingcorresponding to a result of the detection on the image to bereproduced. In particular, if the above-mentioned image cropping isperformed on the image to be reproduced, this example may be used.

In addition, in the image sensing apparatus according to an embodimentof the present invention 1, a control unit such as a microcomputer mayperform the operations of the image processing portion 6, the trackingprocess portion 60 and the like. Further, the whole of a part of thefunctions realized by the control unit may be described as a program,which is executed by a program executing unit (e.g., a computer), sothat the whole or a part of the functions is realized.

In addition, without limiting to the above-mentioned cases, the imagesensing apparatus 1 of FIG. 1 and the tracking process portion 60 ofFIG. 2 can be realized by hardware or a combination of hardware andsoftware. In addition, if the image sensing apparatus 1 and the trackingprocess portion 60 are realized by using software, the block diagram ofthe portion realized by software indicates the functional block diagramof the portion.

Although the embodiments of the present invention are described above,the present invention is not limited to the embodiments, which can bemodified variously within the scope of the present invention withoutdeviation from the spirit thereof.

The present invention can be applied to an image processing apparatusfor detecting a track object from an input image, and an electronicapparatus such as an image sensing apparatus or a reproduction apparatushaving the image processing apparatus.

1. An image processing apparatus comprising: a track value settingportion which sets a track value which is a signal value indicating atrack object in an input image; an auxiliary track value setting portionwhich sets an auxiliary track value which indicates the track object andis a signal value different from the track value; and a track targetdetection portion which detects a pixel having the track value from theinput image, wherein the track value setting portion is capable ofperforming a switching operation of setting the auxiliary track value asa new track value instead of the set track value.
 2. An image processingapparatus according to claim 1, further comprising: a search areasetting portion which sets a search area in the input image; and animage analysis portion which determines a frequency distribution of apixel signal value in the search area, wherein the auxiliary track valuesetting portion sets the auxiliary track value based on the frequencydistribution, the track value setting portion decides whether the trackvalue is correct or not based on the frequency distribution and performsthe switching operation if it is decided that the set track value isincorrect, and the track target detection portion detects a pixel havingthe track value from the image in the search area.
 3. An imageprocessing apparatus according to claim 2, wherein the auxiliary trackvalue setting portion sets a signal value having a higher frequency inthe frequency distribution with higher priority as the auxiliary trackvalue.
 4. An image processing apparatus according to claim 2, whereinthe input image is supplied to the image processing apparatussequentially, so that the image analysis portion outputs the frequencydistribution sequentially, and the auxiliary track value setting portionsets a signal value having a frequency above a predetermined thresholdvalue a predetermined times successively with respect to the frequencydistribution output sequentially as the auxiliary track value.
 5. Animage processing apparatus according to claim 2, wherein the auxiliarytrack value setting portion sets the auxiliary track value by excludinga signal value indicating a background of the input image.
 6. An imagesensing apparatus comprising: an image sensing portion which generatesan input image by image sensing; and an image processing apparatusaccording to claim 1, wherein the image sensing apparatus performscontrol based on a result of the detection of the track object by theimage processing apparatus.